Mark information recording apparatus, mark information presenting apparatus, mark information recording method, and mark information presenting method

ABSTRACT

A mark information recording apparatus includes an output unit and a recording unit. The output unit is configured to output, for displaying a partial image being a part of an entire image having an arbitrary resolution among entire images generated at a plurality of different resolutions with respect to a single target, the partial image. The recording unit is configured to associate mark information with information on a position within the entire image and information on the resolution of the output partial image and record the mark information.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/637,921, filed on Jun. 29, 2017, which is a continuationU.S. patent application Ser. No. 14/744,897, filed on Jun. 19, 2015,which is a continuation of U.S. patent application Ser. No. 13/550,237,filed on Jul. 16, 2012, which claims priority to Japanese PriorityPatent Application JP 2011-159541 filed in the Japan Patent Office onJul. 21, 2011, the entire content of which is hereby incorporated byreference.

BACKGROUND

The present application relates to a mark information recordingapparatus that records electronic mark information, to a markinformation presenting apparatus that presents recorded markinformation, and to a mark information recording method and a markinformation presenting method.

Japanese Patent Application Laid-open No. 2005-236906 (hereinafter,referred to as Patent Document 1) describes a shooting apparatus thatassociates, when shooting a moving image, an electronic mark (essencemark) expressing an index relating to a video with a video frame. Withthis, when editing the video, the user is able to view the frameassociated with the electronic mark as a thumbnail and find out adesired video scene (e.g., see paragraphs [0026] and [0030] in thespecification of Patent Document 1).

SUMMARY

It is desirable to add electronic mark information to not only a videobut also an image.

In view of the above-mentioned circumstances, there is a need for a markinformation recording apparatus that records electronic mark informationto an image, a mark information presenting apparatus capable ofpresenting recorded mark information, and a mark information recordingmethod and a mark information presenting method.

According to an embodiment of the present application, there is provideda mark information recording apparatus including an output unit and arecording unit.

The output unit is configured to output, for displaying a partial imagebeing a part of an entire image having an arbitrary resolution amongentire images generated at a plurality of different resolutions withrespect to a single target, the partial image.

The recording unit is configured to associate mark information withinformation on a position within the entire image and information on theresolution of the output partial image and record the mark information.

The mark information is recorded while being associated with theposition information of the partial image output by the output unit andthe information on the resolution of the partial image (resolution ofthe entire image including the partial image). Therefore, the markinformation recording apparatus is capable of adding the markinformation depending on the position of the partial image. Further, themark information recording apparatus is also capable of adding the markinformation depending on the resolution in addition to the position ofthe partial image.

The concept of “associating A with B” includes both the meanings of“associating A with ID identifying B” and “associating ID identifying Awith B or ID identifying B.”

The recording unit may be configured to record, as the mark information,annotation information and a mark image to be combined in the partialimage.

The mark information recording apparatus may further include anannotation information generation unit configured to start, wheninformation on an input operation by a user for starting generation ofthe annotation information is acquired, the generation of the annotationinformation. With this configuration, according to the information onthe operation by the user, it is possible to record the annotationinformation when the user desires.

Alternatively, the mark information recording apparatus may furtherinclude an annotation information generation unit configured to start,when information on an audio of a speech of a user is acquired,generation of the annotation information. With this configuration, usingthe start of the speech of the user as a trigger, it is possible tostart recording of the mark information.

The mark information recording apparatus may further include anacquisition unit configured to acquire information on an operation by auser for causing the output unit to output the partial image. In thiscase, the recording unit is configured to associate the mark informationwith the position information and the resolution information on thepartial image output by the output unit according to the operationinformation acquired by the acquisition unit. With this configuration,it becomes possible to record the mark information associated with thepartial image output according to the operation by the user (e.g., thepartial image of the entire image, which the user desires to observe).

The recording unit may be configured to associate position informationand resolution information with each of partial images different in atleast one of the position information and the resolution information,the partial images being output by the output unit according toinformation on a series of continuous or intermittent operations by theuser that is acquired by the acquisition unit. In this case, therecording unit is configured to record information obtained by theassociation as output history information on each of the partial imagesand associate the mark information with the history information. Withthis configuration, it is possible to record the mark information foreach piece of history information on the series of operations by theuser for continuously outputting the plurality of different partialimages.

The recording unit may be configured to execute, when, out of theoperation information acquired by the acquisition unit, predeterminedoperation information is acquired, the association processing of themark information with respect to the partial image output according tothe predetermined operation information. With this configuration, when,out of various types of operation information by a user for outputtingthe partial image, the limited and predetermined operation informationis acquired by the acquisition unit, it is possible to associate themark information with the partial image output by the operationinformation.

The recording unit may be configured to associate priority informationindicating a priority upon presentation of each of a plurality of piecesof mark information with each of the plurality of pieces of markinformation and record the priority information. With thisconfiguration, according to the set priority, it is possible to presentat least one of the plurality of pieces of mark information.

The recording unit may be configured to associate, with each of at leasttwo or more resolution ranges including the resolution of the partialimage output by the output unit, the mark information. In other words,the recording unit is capable of executing, for each resolution rangeincluding the resolution of the output partial image, the associationprocessing of the mark information.

The recording unit may be configured to set a threshold of theresolution according to an operation by the user and to associate theset threshold of the resolution with the position information and themark information and record the set threshold. With this configuration,upon presentation of the mark information, for example, when themagnification of the partial image to be output becomes equal to orlarger than the threshold, it is possible to present the markinformation.

According to an embodiment of the present application, there is provideda mark information presenting apparatus including an output unit, arecording unit, and a processing unit.

The output unit is configured to output, for displaying a partial imagebeing a part of an entire image having an arbitrary resolution amongentire images generated at a plurality of different resolutions withrespect to a single target, the partial image.

The recording unit is configured to associate mark information withinformation on a position within the entire image and information on theresolution of the output partial image and record the positioninformation and the resolution information that are associated with themark information as recorded position information and recordedresolution information, respectively.

The processing unit is configured to execute, when an output positioninformation group constituting the partial image to be output includesthe recorded position information, processing for presenting the markinformation associated with at least the recorded position information.

According to the embodiment of the present application, when the outputposition information group constituting the partial image to be outputin the entire image includes the recorded position information, it ispossible to present the mark information associated with the recordedposition information.

The processing unit may be configured to execute, when the outputposition information group includes the recorded position informationand a resolution of output resolution information of the partial imageto be output by the output unit is equal to a resolution of the recordedresolution information corresponding to the recorded positioninformation, processing for presenting the mark information associatedwith the recorded position information and the recorded resolutioninformation. With this configuration, it is possible to present the markinformation associated with both of the recorded position informationand the recorded resolution information on the partial image.

The recording unit may be configured to record, as the mark information,annotation information and a mark image to be combined in the partialimage. In this case, the processing unit is configured to executeprocessing for presenting the annotation information.

The processing unit may be configured to combine, when the outputposition information group constituting the partial image to be outputincludes a plurality of pieces of recorded position information, aplurality of mark images associated with the plurality of pieces ofrecorded position information, in the partial image to be output. Withthis configuration, when the plurality of pieces of mark information areassociated with the single output partial image, it is possible for adisplay unit connected to this mark information presenting apparatus todisplay the plurality of mark images associated with the pieces of markinformation.

The processing unit may be configured to combine, among a plurality ofpieces of recorded resolution information associated with the pluralityof pieces of recorded position information, one of an image having aresolution equal to the resolution of the output resolution informationof the partial image to be output by the output unit and an imagecorresponding to resolution information including a resolution within arecorded resolution range, in the partial image to be output.

The recording unit may be configured to associate priority informationindicating a priority upon presentation of each of a plurality of piecesof annotation information corresponding to the plurality of images witha plurality of pieces of mark information and record the priorityinformation. In this case, the processing unit is configured to set,according to the recorded priority information, one of whether or not topresent the plurality of pieces of annotation information and a timeorder of presenting the plurality of pieces of annotation information.With this configuration, it is possible to present the pieces ofannotation information to the user in a descending priority order.

The recording unit may be configured to associate, with each of at leasttwo or more resolution ranges including the resolution of the outputresolution information of the partial image output by the output unit,the annotation information and set a different recorded resolution rangefor each of different pieces of priority information. In this case, theprocessing unit is configured to execute processing for presenting atleast one piece of annotation information among a plurality of pieces ofannotation information according to the set recorded resolution range,the plurality of pieces of annotation information being associated withthe plurality of pieces of recorded position information included in theoutput position information group.

The mark information presenting apparatus may further include anacquisition unit configured to acquire information on an operation by auser for causing the output unit to output the partial image. In thiscase, the recording unit is configured to execute, when, out of theoperation information acquired by the acquisition unit, predeterminedoperation information is acquired, the association processing of themark information with respect to the partial image output according tothe predetermined operation information. Further, the processing unit isconfigured to execute, when an output position information groupconstituting the partial image output according to the predeterminedoperation information includes the recorded position information,processing for presenting the mark information associated with at leastthe recorded position information.

According to an embodiment of the present application, there is provideda mark information recording method including outputting, for displayinga partial image being a part of an entire image having an arbitraryresolution among entire images generated at a plurality of differentresolutions with respect to a single target, the partial image.

Mark information is recorded while being associated with information ona position within the entire image and information on the resolution ofthe output partial image.

According to an embodiment of the present application, there is provideda mark information presenting method including outputting, fordisplaying a partial image being a part of an entire image having anarbitrary resolution among entire images generated at a plurality ofdifferent resolutions with respect to a single target, the partialimage.

Mark information is associated with information on a position within theentire image and information on the resolution of the output partialimage and the position information and the resolution information thatare associated with the mark information are recorded as recordedposition information and recorded resolution information, respectively.

When an output position information group constituting the partial imageto be output includes the recorded position information, processing forpresenting the mark information associated with at least the recordedposition information is executed.

As described above, according to the embodiments of the presentapplication, it is possible to record mark information to an image.Further, it is possible to present recorded mark information.

These and other objects, features and advantages of the presentdisclosure will become more apparent in light of the following detaileddescription of best mode embodiments thereof, as illustrated in theaccompanying drawings.

Additional features and advantages are described herein, and will beapparent from the following Detailed Description and the figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram showing a configuration of hardware of a PC,for example, as an information processing apparatus including a markinformation recording apparatus and a mark information presentingapparatus according to an embodiment of the present application;

FIG. 2 is a diagram showing an image pyramid structure for describing aprinciple of displaying images of the image pyramid structure;

FIG. 3 is a diagram for describing a procedure for generating the imagegroup of the image pyramid structure;

FIG. 4 is a diagram showing an example of an input unit of the PC;

FIG. 5 is a diagram showing size examples of an entire image and apartial image;

FIG. 6 is a flowchart showing record processing according to a firstembodiment of record processing;

FIG. 7 is a table created by the record processing according to thefirst embodiment;

FIG. 8A shows an example of a screen displayed on a display unit whenaudio information is recorded;

FIG. 8B shows a modified example of the screen;

FIG. 9 is a flowchart showing record processing according to a secondembodiment;

FIG. 10 is a flowchart showing record processing according to a thirdembodiment;

FIG. 11 is a flowchart showing record processing according to the thirdembodiment;

FIG. 12 is a flowchart showing record processing according to a fifthembodiment;

FIG. 13 is a table of history information that is created by the recordprocessing according to the fifth embodiment;

FIG. 14 is a diagram showing movement of a plurality of partial imagesregarding the history information in the fifth embodiment;

FIG. 15 is a table created by record processing according to a sixthembodiment;

FIG. 16 is a table created by record processing according to an eighthembodiment;

FIG. 17 is a flowchart according to a first embodiment of presentationprocessing;

FIGS. 18A and 18B are diagrams each showing a partial image in which amark image is combined, which is obtained by the presentation processingaccording to the first embodiment;

FIG. 19 is a flowchart showing presentation processing according to asecond embodiment;

FIG. 20 is a diagram showing a partial image in which a mark image iscombined, which is obtained by presentation processing according to athird embodiment;

FIG. 21 is a diagram showing a partial image in which a mark image iscombined, which is obtained by the presentation processing according tothe third embodiment;

FIG. 22 is a flowchart showing presentation processing according to afourth embodiment;

FIGS. 23A and 23B are diagrams each showing a partial image in which amark image is combined, which is obtained by presentation processingaccording to a fifth embodiment;

FIG. 24 is a diagram showing a partial image in which a mark image iscombined, which is obtained by the presentation processing according tothe fifth embodiment;

FIG. 25 is a table created by the record processing according to theabove-mentioned third embodiment, which is used in presentationprocessing according to a sixth embodiment;

FIG. 26 is a flowchart showing presentation processing according to aseventh embodiment; and

FIG. 27 is a flowchart showing the presentation processing according tothe seventh embodiment.

DETAILED DESCRIPTION

Hereinafter, embodiments of the present application will be describedwith reference to the drawings.

FIG. 1 is a block diagram showing a configuration of hardware of a PC(personal computer), for example, as an information processing apparatusincluding a mark information recording apparatus and a mark informationpresenting apparatus according to an embodiment of the presentapplication.

A PC 100 includes a CPU (central processing unit) 101, a ROM (read onlymemory) 102, a RAM (random access memory) 103, an input/output interface105, and a bus 104 that connects them to one another.

To the input/output interface 105, a display unit 106, an input unit107, a storage unit 108, a communication unit 109, a drive unit 110, andthe like are connected.

The display unit 106 is a display device using, for example, a liquidcrystal or an EL (electro-luminescence).

The input unit 107 includes, for example, a pointing device, a keyboard,a touch panel, a microphone, and other operation apparatuses. In thecase where the input unit 107 includes a touch panel, this touch panelmay be integrated with the display unit 106.

FIG. 4 is a diagram showing an example of the input unit 107. This inputunit 107 has shape and functions similar to those of a game controller 1and includes arrow keys 2 and push buttons A, L, and R, and the like.The arrow keys 2 are, for example, keys for the user to select aspecified area (display area) within an entire image in order to specifyand display a partial image being a part of the entire image, which willbe described later. The push buttons L and R have enlarging anddownsizing functions for changing an observation magnification of thepartial image. Further, the push buttons A have functions of startingand stopping creation processing of annotation information, which willbe described later, or other functions.

The storage unit 108 is a non-volatile storage device and includes, forexample, an HDD (hard disk drive), a flash memory, and other solid-statememories.

The drive unit 110 is a device capable of driving a removable recordingmedium 111 such as an optical recording medium, a floppy (registeredtrademark) disk, a magnetic recording tape, and a flash memory. Incontrast, the above-mentioned storage unit 108 is often used as a deviceinstalled into the PC 100 in advance for mainly driving a non-removablerecording medium.

The communication unit 109 includes a modem, a router, and othercommunication devices that are connectable to a LAN (local areanetwork), a WAN (wide area network), and the like and communicate withother devices. The communication unit 109 may perform a wired orwireless communication. The communication unit 109 is often usedseparately from the PC 100.

Next, an image obtained by an optical microscope (not shown) to bemainly stored in the storage unit 108 of the PC 100 and a principle ofdisplaying the image will be described. FIG. 2 is a diagram showing animage pyramid structure for describing the displaying principle.

An image pyramid structure 50 in this embodiment is an image group(entire image group) generated at a plurality of different resolutionswith respect to a single image obtained by the optical microscope from asingle observation target 15 (see FIG. 3). At the bottom of the imagepyramid structure 50, a largest-size image is provided. At the top ofthe image pyramid structure 50, a smallest-size image is provided. Theresolution of the largest-size image is, for example, 50*50 (kpixels:kilo pixels) or 30*40 (kpixels) as shown in FIG. 5. The resolution ofthe smallest-size image is, for example, 256*256 (pixels) or 256*512(pixels).

Specifically, for example, when the single display unit 106 displaysthose images at 100% (displays each image with physical dots as many asthe number of pixels of the image), the largest-size image is displayedas a largest one and the smallest-size image is displayed as a smallestone. Here, in FIG. 2, the display ranges of the display unit 106 areindicated by D.

FIG. 3 is a diagram for describing a procedure for generating the imagegroup of the image pyramid structure 50.

First, a digital image of an original image (huge image) obtained by theoptical microscope (not shown) at a predetermined observationmagnification is prepared. This original image corresponds to thelargest-size image being the image at the bottom of the image pyramidstructure 50 shown in FIG. 2. That is, the original image is an imagehaving a highest resolution. Therefore, as the image at the bottom ofthe image pyramid structure 50, an image observed and obtained by theoptical microscope at a relatively high magnification is used.

It should be noted that, in the field of pathology, one obtained byslicing an organ, a tissue, a cell of a living body, or a part of themgenerally becomes the observation target 15. Then, a scanner apparatus(not shown) having functions of the optical microscope scans theobservation target 15 sandwiched between glass slides and the digitalimage thus obtained is stored in this scanner apparatus or anotherstorage apparatus.

The scanner apparatus or a general-purpose computer (not shown)generates a plurality of images from the largest-size image, which hasbeen obtained in the above-mentioned manner, by gradually decreasing theresolution as shown in FIG. 3 and stores the generated images in, forexample, a “tile” unit being a predetermined size unit. The size of asingle tile is, for example, 256*256 (pixels). The image group thusgenerated forms the image pyramid structure 50 and this image pyramidstructure 50 is stored in the storage unit 108 of the PC 100. Actually,the PC 100 only needs to store the images having a plurality ofdifferent resolutions and information on the resolutions whileassociating each image with information on its resolution. It should benoted that the generation and storage of the image pyramid structure 50may be executed by the PC 100 shown in FIG. 1.

The entire image group that forms the image pyramid structure 50 may begenerated by a known compression method or may be generated by a knowncompression method applied for generating a thumbnail image, forexample.

Using software adopting the system of the image pyramid structure 50,the PC 100 extracts, according to information on an input operation bythe user via the input unit 107, a desired image from the image pyramidstructure 50 and outputs this image to the display unit 106.Specifically, the PC 100 displays an image of an arbitrary part of animage having an arbitrary resolution selected by the user, the image ofthe arbitrary part being specified by the user. In this case, at leastone of the CPU 101, the input/output interface 105, and the display unit106 functions as an output unit that outputs the image of the arbitrarypart to the outside for displaying it on the display unit 106. Further,the CPU 101 or the input/output interface 105 functions as anacquisition unit that acquires the information on the input operation bythe user.

For example, as shown in FIG. 5, an image 210 being a part of an entireimage 200 having an arbitrary resolution among the entire images havinga plurality of different resolutions is specified by an input operationby the user. The PC 100 includes the entire images 200 having aplurality of different resolutions as described above. When the inputoperation by the user specifies the image 210, the PC 100 is able to cutout the image in the specified area as the partial image 210 and tooutput the partial image 210 to the display unit 106.

In the example shown in FIG. 5, the partial image 210 is cut out fromthe largest-size original image and the display unit 106 displays thecut-out partial image 210 on the entire screen of the display unit 106,for example, using HD (high definition).

The display unit 106 is capable of displaying the partial image 210having an arbitrary resolution. The resolution of the partial image 210depends on the resolution of the entire image being the original imageto be cut out. As a matter of course, the display unit 106 may displaythe cut-out partial image in a part of the entire screen of the displayunit 106.

The above-mentioned processing enables the user to feel as if the userobserves the observation target 15 while changing the observationmagnification. In other words, the PC 100 functions as a virtualmicroscope. Here, the virtual observation magnification corresponds tothe resolution in reality.

[Record Processing of Mark Information by PC as Mark InformationRecording Apparatus]

The present application is a application of associating mark informationwith information on a position (coordinate) of the partial image 210within the entire image and information on its resolution, the partialimage 210 being to be output to the display unit 106.

Hereinafter, record processing of the mark information by the PC 100will be described. The mark information is a symbol, a sign, an index,annotation information, or an image expressing them.

The annotation information is information constituted of a text, anaudio, an image, a link (e.g., URL: uniform resource locator), or thelike, and may be associated with the above-mentioned symbol, sign,index, and the like.

Hereinafter, on the assumption that the mark information is an image tobe combined in the annotation information (in particular, audioinformation constituted of speech of user) and the partial image 210,the record processing thereof will be described. That is the case with adescription of “presentation processing of mark information.”

It should be noted that the processing by the PC 100, which will bedescribed later, is realized by a cooperation of software stored in thestorage unit 108, the ROM 102, or the like with a hardware resource ofthe PC 100. Specifically, by the CPU 101 loading programs that configurethe software stored in the storage unit 108, the ROM 102, and the likeinto the RAM 103 and executing the programs, the following processing isrealized. For the sake of description, on the assumption that a subjectthat executes the processing is the CPU 101, the processing will bedescribed. That is the case with the description of “presentationprocessing of mark information.”

First Embodiment of Record Processing

FIG. 6 is a flowchart showing record processing according to a firstembodiment.

According to an input operation by the user using the arrow keys 2 ofthe controller 1, for example, the CPU 101 performs pan, zoom-in, andzoom-out within the entire image 200, and the user searches for thepartial image 210 that the user has been observed. Here, pan refers to amovement of a specified area (display area) in the display unit 106, themovement having a component in at least one direction of horizontal andvertical directions.

Hereinafter, at least one processing of pan, zoom-in, and zoom-outwithin the entire image 200 by the CPU 101 is referred to asspecification processing. Specifically, the CPU 101 outputs, every timethe CPU 101 executes the specification processing, the partial image 210to the display unit 106. Although described above, the specified arearefers to an area constituting the partial image 210 within the entireimage 200, the partial image 210 being to be output to the display unit106 by the specification processing.

The CPU 101 executes the specification processing according to the inputoperation by the user as described above. Then, when the CPU 101displays the partial image 210 that the user desires to observe, theuser pushes, using the controller 1, the push button A serving as arecording button. In response to this operation, the CPU 101 starts therecord processing of an audio (Step 101). The CPU 101 increments an IDthat identifies recorded audio information (Step 102). Further, the CPU101 stores, in a table shown in FIG. 7, information on a position(coordinate) of the partial image 210 within the entire image 200 andits observation magnification (hereinafter, simply referred to asmagnification) while associating them with the ID (Step 103), thepartial image 210 being output for display when the record processing isstarted. Then, the CPU 101 starts processing of recording the audioinformation on the speech of the user, which is input via themicrophone, in the storage unit 108 or the like (Step 104). In thiscase, the CPU 101 functions as an annotation information generationunit.

The CPU 101 records the audio information while associating the audioinformation with (the ID, the position information, and the resolutioninformation on) the partial image 210. In this case, the CPU 101functions as a recording unit.

A coordinate (representative coordinate) indicating the partial image210 recorded in the table only needs to be a coordinate at apredetermined position of a coordinate group constituting the partialimage 210. For example, the coordinate is a coordinate at a centerposition, a coordinate at a most-upper left position, or a coordinate atanother predetermined position of the partial image 210.

While the push button A is being pushed (No in Step 105), the CPU 101continues the record processing (Step 106). When the user releases thefinger from the push button A, the CPU 101 terminates the recordprocessing of the audio (Step 107).

FIG. 8A shows an example of a screen displayed on the display unit 106when the audio information is recorded. When the record processing ofthe audio information is started or terminated, the CPU 101 combines, inthe partial image 211 recorded in, for example, the table, a mark imageM identifying the annotation information and associates the mark image Mwith the ID of the partial image 211 as shown in FIG. 7. Forminformation on the mark image M shown in FIGS. 7 and 8A is informationof blue and circle shape.

As the form of the mark image M, other colors and shapes may be adopted.Further, the form of the mark image M may vary depending on each user.Alternatively, the form of the mark image M may vary depending onwhether it indicates the audio information or other annotationinformation (e.g., link image or text information), i.e., depending onan annotation information type. Alternatively, the form of the markimage M may vary depending on a priority, which will be described later.

The CPU 101 combines the mark image M at a predetermined position, e.g.,a center position in the partial image 211. A coordinate of the positionat which the mark image M is combined may be the above-mentionedcoordinate indicating the partial image 210 (representative coordinaterecorded in table).

Alternatively, as shown in FIG. 8B, the mark image M may be combined ata coordinate of an arbitrary position in the partial image 210, that theuser specifies with a pointer (or by a touch operation with respect tothe screen). In this case, coordinate information on the mark image Mmay be also stored in the table shown in FIG. 7. The audio informationmay be associated with the coordinate of the mark image M.

As one of methods of presenting (e.g., reproducing) the recorded audioinformation to the user, the following processing is contemplated. Forexample, when, with respect to the position of the mark image M thuscombined, the user makes an input operation such as a click (or touch),the CPU 101 reproduces the recorded audio information as will bedescribed later. In this case, the mark image M functions a GUI(graphical user interface).

As described above, in this embodiment, the annotation information beingthe mark information is stored while being associated with the positioninformation on the partial image 211 that has been output for display,and the information on the resolution of the partial image 211(resolution of the entire image 200 including the partial image 211).Therefore, the CPU 101 is capable of adding the annotation informationbased on the coordinate of the partial image 210. Further, the CPU 101is capable of adding the annotation information based on not only thecoordinate of the partial image 210 but also the magnification thereof.

Further, in this embodiment, since the record processing is startedbased on the information on the operation made by the user with respectto the controller 1, it is possible for the user to record the audioinformation when the user desires.

In the above description, when the user releases the finger from thepush button A, the record processing is terminated. However, the recordprocessing may be executed using a toggle function. Specifically, forexample, when the user may first push the push button A, the recordingmay be started, and when the user may re-push the same push button A,the recording may be terminated. Alternatively, using separate pushbuttons, the start and end of the recording may be executed.

Second Embodiment of Record Processing

FIG. 9 is a flowchart showing record processing according to a secondembodiment. Regarding the following embodiments, descriptions of theprocessing same as that in the first embodiment described above will besimplified or omitted.

In this embodiment, the start of the speech of the user is detected bythe microphone (not shown) (input unit 107) and the detection is set asa trigger for starting the record processing.

According to an input operation by the user, the CPU 101 performs thespecification processing and displays the partial image that the userdesires to observe. Here, when the speech of the user occurs (Yes inStep 201), the ID is incremented (Step 202). The coordinate and theinformation on the magnification of the partial image 210 beingdisplayed on the display unit 106 are stored in the table (see FIG. 7)while being associated with the ID (Step 203). In Step 201, for example,when an input of an audio having a sound pressure level above athreshold is made, the CPU 101 may determine that the speech of the useroccurs.

When a silent period continues for a predetermined period of time (Yesin Step 205), the CPU 101 terminates the record processing (Step 207).The silent period is set as a period in which, for example, an input ofan audio having a sound pressure level equal to or smaller than theabove-mentioned threshold is made.

As described above, not using the conscious input operation by the userwith respect to the controller 1, but using the start of the speech ofthe user as a trigger, it is possible to start the record processing ofthe audio information. With this, operability for the user is enhancedand it is possible to easily record an audio.

The PC 100 may include a program by which a predetermined word(s), forexample, “start recording” or “start,” in the speech of the user isregistered in advance and when the user says this word, Yes isdetermined in Step 201 using this word as a trigger. In this case, alsoregarding Step 207, the PC 100 may include a program by which when theuser says a registered word, the processing of Step 207 is executedusing this word as a trigger.

Third Embodiment of Record Processing

FIGS. 10 and 11 are flowcharts each showing record processing accordingto a third embodiment.

In this embodiment, when, in the middle of recording the speech of theuser, the user makes a predetermined input operation with respect to theinput unit 107, the coordinate and the like of the partial image 210 tobe output by the input operation are recorded according to informationon the input operation.

As shown in FIG. 10, when the speech of the user occurs (Yes in Step301), the CPU 101 starts the record processing of the audio of thespeech (Step 302). When the silent period does not continue for thepredetermined period of time (No in Step 303), the CPU 101 records theaudio information in a temporary buffer being a kind of memory includedin the PC 100 (Step 304). When the silent period continues for thepredetermined period of time (Yes in Step 303), the CPU 101 terminatesthe record processing of the audio (Step 305).

Criteria for the determinations in Steps 301 and 303 are the same asthose in Steps 201 and 205 described above, respectively.

Subsequently, with a monitoring loop of an audio input state as shown inFIG. 10 being executed, a monitoring loop of an input operation state ofthe user is executed as shown in FIG. 11.

The CPU 101 determines whether or not a zoom-in (enlargement) operationof the screen is started as the predetermined input operation by theuser (Step 401). When the zoom-in operation is started, the CPU 101determines in the monitoring loop processing of the audio input stateshown in FIG. 10 whether or not the audio is being recorded (Step 402).When the audio is not being recorded (No in Step 402) and the zoom-inoperation is terminated (Yes in Step 403), the CPU 101 returns to Step401.

When, in Step 402, the audio is being recorded, the CPU 101 incrementsthe ID (Step 404) and stores the coordinate and the magnification of thepartial image 210, which is being output for display, together with theID in the table shown in FIG. 7 (Step 405).

The CPU 101 determines in the monitoring loop processing of the audioinput state shown in FIG. 10 whether or not the audio is being recorded(whether or not the record processing of the audio is terminated) (Step406). When the record processing is terminated, the CPU 101 stores theaudio information, which has been temporarily stored in the temporarybuffer, in the table shown in FIG. 7 while associating the audioinformation with the ID and the like (Step 407).

In this embodiment, the CPU 101 only needs to record, in Step 407, theaudio information temporarily stored from Step 401 to Step 406.Alternatively, the CPU 101 only needs to record the audio informationtemporarily stored from a time at which a predetermined time has elapsed(e.g., one to ten seconds) after Step 401 to Step 406. Alternatively,the CPU 101 may record the audio information temporarily stored frombefore Step 401 to Step 406.

In this embodiment, when the user has made a zoom-in operation of inputoperations and the record processing of the audio is being executed, thepartial image 210 displayed by the zoom-in operation may be recorded.When a zoom-in operation is made, the partial image 210 displayed by thezoom-in often includes an important point. Therefore, the processingaccording to this embodiment has a great merit.

The predetermined input operation to be a determination target in Steps401 and 403 is not limited to the zoom-in operation, and may be apointer or touch motion, for example, pan or zoom-out. Further, the PC100 may include a program that allows the user to customize and set thepredetermined input operation to be such a determination target.

Fourth Embodiment of Record Processing

Although not shown in the figures, the following processing may berealized.

For example, if the CPU 101 does not execute the monitoring loop of theaudio input state as shown in FIG. 10, the CPU 101 may start the recordprocessing immediately after a predetermined input operation (e.g.,zoom-in operation as described above) by the user is started or after apredetermined time (e.g., one to ten seconds) has elapsed from the startof the predetermined input operation. In this case, a method ofterminating the recording may be the method shown in FIG. 7 or 9.

Fifth Embodiment of Record Processing

FIG. 12 is a flowchart showing record processing according to a fifthembodiment.

The processing of this embodiment is processing in which, as shown inFIG. 14, a plurality of partial images 210 output by a series ofcontinuous or intermittent input operations by the user are recorded andthe partial images 210 are associated with audio information. FIG. 14shows an example in which a series of input operations by the usersuccessively outputs images of specified areas 1 to 5. Regarding thespecified areas 1 to 4, they have the same magnification and the partialimages thereof have different coordinates. A center coordinate of thespecified area 5 is the same as that of the specified area 4, and thespecified areas 4 and 5 have a plurality of different magnifications(the specified area 5 has a higher magnification).

For example, when the speech of the user occurs, the CPU 101 incrementsthe ID and starts the record processing of the audio (Step 501 to 503).

Here, the CPU 101 associates coordinate and resolution information oneach of the plurality of different partial images 210 output accordingto information on the series of continuous or intermittent inputoperations by the user with the partial image 210 and records them ashistory information on the specification processing (Step 504). FIG. 13shows a table of the history information on the specificationprocessing. Thus, the plurality of different partial images and theaudio information are associated with a single ID.

The plurality of different partial images mean partial images differentin at least one of the coordinate and the resolution.

When a silent period continues for a predetermined period of time (Yesin Step 506), the CPU 101 terminates the record processing (Step 507).

The information on the series of continuous or intermittent inputoperations refers to information on operations made by the user at atime interval within a predetermined time, which is set in advance orcustomizable by the user. Specifically, the time interval within thepredetermined time refers to, if the user performs single pan within theentire image 200 and performs pan again, a time interval when no inputoperations for specifying an area are made by the user between the twopan operations.

The predetermined time is, for example, equal to or less than onesecond. Referring to FIG. 14, it is a time interval when the partialimage 210 does not move between pan of the specified areas 1 to 2 andpan of the specified area 2 to 3, for example.

As described above, in this embodiment, for each piece of historyinformation on the series of operations by the user for continuouslyoutputting the plurality of different partial images 210, the audioinformation may be recorded.

In this embodiment, setting the occurrence of the speech as a trigger,the record processing is started. However, as shown in FIG. 7, when theuser pushes the recording button, the record processing may be startedand the history information may be associated with the audioinformation.

Sixth Embodiment of Record Processing

FIG. 15 is a table created by record processing according to a sixthembodiment.

In this embodiment, the CPU 101 associates each of IDs of a plurality ofpieces of audio information being a plurality of pieces of annotationinformation with priority information indicating a priority forpresenting (reproducing) the audio information. Thus, the CPU 101 iscapable of reproducing at least one of the pieces of audio informationin accordance with the set priority. The reproduction of the audioinformation according to the priority will be described in “presentationprocessing” below in detail.

Seventh Embodiment of Record Processing

In each of the above-mentioned embodiments, the partial image 210 thathas been output for display on the display unit 106 has a singlemagnification value associated with the audio information in the table.However, upon recording of the audio information, the audio informationmay be associated with each of at least two or more magnification values(i.e., magnification ranges) including a magnification value of thepartial image 210 that has been output for display on the display unit.

In other words, upon recording of the audio information, the CPU 101associates the audio information with each magnification range includingthe magnification value of the partial image 210 that has been outputfor display. For example, the magnification of ID=1 shown in FIG. 7 is20* magnification. It is the magnification of the partial image 210output when the record processing of the audio information is started.In this case, in the table, a magnification range including 20*magnification, for example, a magnification range of 10 to 30*magnification, 5 to 40* magnification, or 20 to 40* magnification isrecorded.

Accordingly, as will be described later, when the recorded audioinformation is reproduced, the CPU 101 is capable of performing thefollowing processing. Specifically, when an image of a specified areaincluding at least one coordinate of a coordinate group constituting thepartial image is output for display at a magnification within itsmagnification range, the CPU 101 is capable of reproducing audioinformation associated with (the ID of) the partial image.

A trigger, a timing, and the like for starting and terminating therecording of the audio information of this embodiment may be the same asany one of those in the above-mentioned embodiments.

This embodiment may be applied to each of the above-mentionedembodiments shown in FIGS. 13 and 15.

Eighth Embodiment of Record Processing

FIG. 16 shows a table created by record processing according to aneighth embodiment.

In this embodiment, for example, the user sets an arbitrarymagnification threshold and the CPU 101 records the set magnificationbeing the magnification threshold in the table. After the table as shownin FIG. 7 or the like has been created, the user may set a magnificationand the CPU 101 may record the set magnification in the table any time.In such a case, the recorded original magnification may be overwritten.

Alternatively, the present disclosure is not limited to the case wherethe user sets a magnification, the CPU 101 may set a magnification inaccordance with a predetermined algorithm, for example, at the start orend or in the middle of the record processing of the audio informationor the like.

According to this embodiment, as will be described later, uponpresentation of the mark information, for example, when a magnificationof the partial image 210 to be output is equal to or larger than a setmagnification, the mark information may be presented.

Other Embodiment of Record Processing

In each of the above-mentioned embodiments, a time stamp at least whenthe record processing has been executed (which may include a concept ofday, month, and year), such as the start or end of the recording of theaudio information, may be also associated with the ID and recorded.

[Presentation Processing of Mark Information by PC as Mark InformationPresenting Apparatus]

Next, a description will be made of processing in which after the tablehas been created in the above-mentioned manner in each of theembodiments, the annotation information is presented by the PC to theuser, the annotation information being one of the pieces of markinformation recorded in the table. When performing this presentationprocessing, the CPU 101 functions as a processing unit.

First Embodiment of Presentation Processing

FIG. 17 shows a flowchart showing processing of presenting (reproducing)audio information as the annotation information based on recorded datain the table shown in FIG. 7, for example. The recorded data is,specifically, a coordinate, a magnification, and annotation informationthat have been recorded in the table. Hereinafter, the coordinate andmagnification recorded in the table are referred to as a recordedcoordinate (recorded position information) and a recorded magnification(recorded resolution information), respectively.

When an input operation by the user for displaying a partial image isbeing executed, the CPU 101 acquires information on the input operation(Yes in Step 601). In this case, the CPU 101 functions as an acquisitionunit.

The CPU 101 compares a coordinate group (output coordinate group asoutput position information group) constituting a partial image of aspecified area specified by the input operation and a magnification(output magnification as output resolution information) of the partialimage with the recorded data in the table (Step 602).

When an output coordinate equal to the recorded coordinate in the tableare present in the output coordinate group constituting the partialimage 210 and the output magnification of the partial image 210 is equalto or larger than the recorded magnification in the table, the CPU 101proceeds to Step 604 (Step 603).

In the case of Yes in Step 603, the CPU 101 executes processing foroutputting audio data associated with an ID corresponding to therecorded magnification to a speaker (not shown) connected to the PC 100(or installed in the PC), that is, reproduction processing (Step 604).

According to this embodiment, even if the CPU 101 combines, as shown inFIG. 18A, the recorded mark image M in the partial image 212 having alower magnification and outputs it (see second embodiment describedbelow), as long as the output magnification is not equal to (or notlarger than) the recorded magnification, the CPU 101 does not reproducethe audio information. When as shown in FIG. 18B, the outputmagnification of the partial image 213 is equal to or larger than therecorded magnification, the audio information is reproduced.

For example, it is also possible to apply the seventh embodiment ofrecord processing described above to this embodiment. Specifically, whenthe output magnification of the partial image of the specified areafalls within the magnification range recorded in the table, the CPU 101is capable of presenting at least one of the mark image M and the audioinformation that are associated with the magnification range. Thisconcept is also applied to first to eight embodiments of presentationprocessing, which will be described later.

Such an operation that the user selects a particular part from a hugeimage being an observation target and the selected part is diagnosed isgenerally performed in a pathology diagnosis. In such circumstances, itis important to record the contents of diagnoses made by particularlyprofessionals for respective partial images and reproduce them for aneducational purpose or the like so that students can study how to makethe diagnoses and the contents. Therefore, a voice memo as the markinformation according to the present application is very useful. Asdescribed above, in the present application, when an input operationcorresponding to a recorded input operation situation of the user (orinput operation near this as will be described later) is made, acorresponding voice memo is reproduced, which is very convenient and hasa great merit particularly in the field of pathology.

Second Embodiment of Presentation Processing

FIG. 19 shows a flowchart showing presentation processing according to asecond embodiment.

In this embodiment, it is assumed that, for example, as shown in FIGS.18A and 18B, the partial image 212 (213) in which the mark image M hasbeen combined is being displayed. In Step 703, when a plurality ofarbitrary coordinates in an output coordinate group of a partial image214 are equal to a single recorded coordinate in the table, that is,when the output coordinate group includes the recorded coordinate, theCPU 101 combines the mark image M corresponding to the recordedcoordinate in the partial image 212 (213) and outputs it (Step 704).

It is assumed that the CPU 101 has acquired information on an inputoperation by the user with respect to the mark image, such as a click ofa GUI being the mark image, for example (Step 705). Then, irrespectiveof the recorded magnifications in the table, i.e., for any outputmagnifications, the CPU 101 reproduces audio information associated withthe mark information operated by the user (Step 706).

According to this embodiment, irrespective of the recordedmagnifications, it is possible to reproduce the audio information by theinput operation with respect to the mark image M.

Third Embodiment of Presentation Processing

By the algorithm described in the first and second embodiments ofpresentation processing above, the CPU 101 is also capable of combining,as shown in FIG. 20, a plurality of mark images M (M1 to M4) in thesingle partial image 214 output by the specification processing andoutputting them. This embodiment is applied to a case where a pluralityof recorded coordinates equal to a plurality of arbitrary coordinates inthe output coordinate group of the partial image 214 and audioinformation each associated with them are present.

In this case, the CPU 101 may combine, in the partial image 214 to beoutput for display, mark images M corresponding to one or more IDs thatinclude the plurality of recorded coordinates in the output coordinategroup constituting the partial image 214 and that have recordedmagnifications higher than an output magnification of the partial image214. With this, the CPU 101 is capable of displaying, when the outputmagnification of the partial image becomes larger than a certainrecorded magnification, the mark image M associated with the ID having arecorded magnification equal to or larger than the recordedmagnification or reproducing audio information associated with the markimage M.

Alternatively, when the plurality of mark images M are being displayedas shown in FIG. 20, if the CPU 101 acquires information on an inputoperation (e.g., click) by the user with respect to one mark image M1 ofthe plurality of mark images M, the CPU 101 may reproduce audioinformation associated with the mark image M1.

Alternatively, the CPU 101 may perform the following processing.Specifically, the CPU 101 determines a correspondence between the outputmagnification and each of the recorded magnifications corresponding tothe plurality of recorded coordinates. In this case, even if the partialimage 214 includes the plurality of recorded coordinates as shown inFIG. 20, the CPU 101 is capable of reproducing audio informationcorresponding to an ID having a recorded magnification equal to theoutput magnification. In this case, for example, when the user makes aninput operation for increasing the magnification of the partial image214, the CPU 101 may combine, as shown in FIG. 21, only a mark image M2corresponding to the recorded magnification equal to the magnificationof a partial image 215, in the partial image 215. In other words, themark images M1, M3, and M4 corresponding to other recordedmagnifications are not displayed.

Fourth Embodiment of Presentation Processing

FIG. 22 shows a flowchart showing presentation processing according to afourth embodiment.

This embodiment is applied to, for example, a case where the outputcoordinate group of the single partial image 210 includes the pluralityof recorded coordinates and the recorded magnifications (or recordedmagnification ranges being recorded resolution ranges) associated withthe plurality of recorded coordinates are equal to each other.

Specifically, in Step 803 shown in FIG. 22, the CPU 101 determineswhether or not the plurality of recorded coordinates equal to aplurality of arbitrary coordinates in the output coordinate group arepresent and the recorded magnifications (or recorded magnificationranges) associated with the plurality of recorded coordinates are equalto each other. In the case of Yes in Step 803, the CPU 101 executesprocessing for presenting at least one of a plurality of pieces of audioinformation associated with the plurality of recorded coordinates inaccordance with priority information recorded in the table in FIG. 15,that is, reproduces the piece of audio information (Step 804).

The CPU 101 uses the priority upon reproduction of the audio informationas an index defined as follows, for example.

(1) Index indicating a reproduction time order of audio information orindex for determining whether or not to reproduce the audio information(2) Index for reproducing audio information associated with even apartial image being displayed at a low magnification

The meaning of “priority” can be diversely construed from user'sperspective. For example, the priority may mean which (reproduction ofaudio information recorded by) user among a plurality of users isprioritized. Alternatively, the priority may mean a priority dependingon importance (e.g., importance from medical viewpoint) of the recordedaudio information or the recorded partial image 210.

The priority is typically set by the user, but may be automatically setby the PC 100. In the case where the PC 100 automatically sets apriority, the priority may be a time-series order of recording the markinformation, for example.

As in (1) above, in the case where the priority is used as the indexindicating the reproduction order and time stamps are recorded in thetable, irrespective of the time stamps, according to a reproductionorder priority, the audio information is reproduced.

In the case where (1) above is applied as priority definition, in Step804, the CPU 101 reproduces pieces of audio information in a descendingpriority order.

In the case where (1) above is applied, the CPU 101 is also capable ofreproducing audio information having a highest priority (or pieces ofaudio information having one or more gradual priorities including ahighest priority) without reproducing other audio information.

In the case where (2) above is applied as the priority definition, theCPU 101 executes the following processing. The CPU 101 records pieces ofaudio information in the table while associating the pieces of audioinformation with magnification ranges including an output magnificationof the partial image 210 and sets different magnification ranges fordifferent priority information of the pieces of audio information.

In this case, in Step 804, the CPU 101 reproduces at least one of theplurality of pieces of audio information associated with the pluralityof recorded coordinates in the output coordinate group, in accordancewith the individually-set recorded magnification ranges described above.In this case, for example, it is possible to set a recordedmagnification range to be larger as the priority becomes higher. Withthis, even in the case of the partial image 210 being displayed at a lowmagnification, as long as the partial image 210 has a high priority,audio information associated with the partial image 210 is reproduced.

As described above, in this embodiment, it is possible to present theaudio information to the user in the descending priority order.

It should be noted that in the fourth embodiment, when the recordedmagnification ranges corresponding to the plurality of recordedcoordinates partially overlap with each other, the mark information maybe presented according to the priority.

Fifth Embodiment of Presentation Processing

As in the fourth embodiment, this embodiment is applied to the casewhere the output coordinate group of the single partial image 210includes the plurality of recorded coordinates and the recordedmagnifications (or recorded magnification ranges) associated with theplurality of recorded coordinates are equal to each other. In this case,as in the fourth embodiment, in accordance with the priority informationshown in FIG. 15, the audio information is reproduced. A application notdescribed in the fourth embodiment is as follows.

In this embodiment, for example, as shown in FIG. 23A, the CPU 101combines, in a partial image 216 having a relatively low outputmagnification, both mark images MG and MB in green and blue. Then, when,increasing the output magnification by an input operation by the user,the output magnification becomes equal to the recorded magnificationcorresponding to the mark images M, as shown in FIG. 23B, the CPU 101combines only the mark image MB in blue, which has a higher priorityamong the two mark images M, in a partial image 217 and outputs it. Atthis time, the CPU 101 reproduces audio information associated with themark image MB in blue.

Alternatively, when the partial image 217 is in the state shown in FIG.23B, in the case where the CPU 101 acquires information on an inputoperation by the user with respect to the mark image MB (e.g., the userclicks it), the CPU 101 may reproduce the audio information.

By the way, if the output coordinate group of the single partial image210 includes the plurality of recorded coordinates, the recordedmagnifications thereof are equal to each other (or recordedmagnification ranges overlap with each other), and further, prioritiesthereof are equal to each other, the CPU 101 may record, upon recording,different mark images M. In the example of the partial image 218 shownin FIG. 24, mark images MR and MY in two different colors (red andyellow) are combined. In this case, for example, the CPU 101 is alsocapable of reproducing audio information of the mark image MR in redprior to the mark image MY in yellow.

Sixth Embodiment of Presentation Processing

Presentation processing according to this embodiment is presentationprocessing when, as shown in FIG. 11, a predetermined input operationbeing one of the input operations by the user for displaying a partialimage is made. When the processing shown in FIG. 11 is made, the CPU 101records data as shown in ID=1 in FIG. 25. In other words, when acquiringinformation on a “zoom-in” operation as information on the predeterminedinput operation, the CPU 101 records a coordinate and a magnification ofthe partial image 210 being the zoom-in target image and associatesaudio information with the coordinate and the magnification of thepartial image 210 and records it.

The presentation processing of the audio information in this case isprocessing as follows. The CPU 101 monitors the information on the inputoperation by the user for displaying the partial image 210 and theoutput coordinate of the partial image 210. When the output coordinategroup constituting the partial image 210 that has been output fordisplay includes the recorded coordinate associated with ID=1 and theinput operation is zoom-in, the CPU 101 reproduces audio informationassociated with this recorded coordinate.

With this, irrespective of the recorded magnification, it is possible toreproduce the audio information when the zoom-in input operation ismade.

For example, for an educational purpose, it is desirable that the user(student) make the operation same as an operation made by a recorder(teacher or the like) to find out an important point in a partial image.For such application, when the user makes an operation for displayingthe partial image, the audio is not reproduced only by touching arecording point on a screen. Only after the user performs zoom-in, theaudio is reproduced and it is determined that the user makes a correctoperation. This configuration is not limited to the medical andpathological fields.

On the other hand, as a condition of reproducing the audio information,only after three conditions including the above-mentioned recordedmagnification are satisfied, the audio information may be reproduced.Specifically, when the output coordinate group of the partial image 210includes the recorded coordinate, the zoom-in operation is made, andfurther its output magnification is equal to the recorded magnification(or recorded magnification range), the CPU 101 may reproduce the audioinformation.

It should be noted that in the example shown in ID=2 in FIG. 25, not“zoom-in” but “pan” is set as the information on the predetermined inputoperation, for example, by user customization. In the case of “pan,” forexample, upon recording, a coordinate and a magnification of the partialimage 210 immediately after the pan operation is terminated (partialimage 210 displayed at end of pan operation) only need to be recorded inthe table. When an input operation other than “zoom-in” and “zoom-out”operations is set as the predetermined input operation, a condition ofreproducing the audio information is not limited to two conditions:correspondence of the recorded coordinate; and correspondence of thepredetermined input operation. As the condition of reproducing the audioinformation, three conditions may be used as described above.

Seventh Embodiment of Presentation Processing

FIG. 26 shows a flowchart showing presentation processing according to aseventh embodiment. The presentation processing according to thisembodiment is processing of reproducing, as shown in FIG. 12, audioinformation associated with history information being information on aseries of continuous or intermittent input operations by the user.

While the user is making an input operation for displaying a partialimage 210, the CPU 101 determines whether or not the output coordinategroup constituting the partial image 210 of the specified area includesat least one recorded coordinate of the plurality of recordedcoordinates in the table and its output magnification is equal to therecorded magnification corresponding to the recorded coordinate (orfalls within recorded magnification range) (Step 903). In the case ofYes in Step 903, the CPU 101 reproduces audio information associatedwith an ID thereof (Step 904).

In other words, in this embodiment, when the information on the partialimage 210, which is obtained by the input operation by the user, isincluded in the history information, the audio information isreproduced.

According to this embodiment, in such a manner that the partial image210 that has been output for display in middle of recording the audio isdisplayed on the display unit 106 by the user making an input operation,it is possible to reproduce audio information associated with thepartial image 210 from the start. That is, even if the partial image(partial image of “coordinate 1” in table shown in FIG. 13) that hasbeen output at the start of the record processing of the audio is notdisplayed upon presentation, the audio is reproduced from the start,which is very convenient.

In this case, when reproducing the audio from the start, the CPU 101 maychange the current screen to the partial image 210 output at the startof the record processing of the audio and display the partial image 210.

Alternatively, in the table shown in FIG. 13, when recording the audio,the CPU 101 may record audio data such that continuous audio data isdivided to correspond to each of coordinates 1, 2, . . . . Accordingly,for example, when the partial image 210 displayed by an input operationby the user corresponds to a middle of the recording of the audio, it ispossible to reproduce the audio from the middle of the recording.

In this embodiment, when the output coordinate group of the partialimage 210 includes the recorded coordinate in the table, the audioinformation may be reproduced irrespective of the recordedmagnification.

Eighth Embodiment of Presentation Processing

FIG. 27 shows a flowchart showing presentation processing according toan eighth embodiment. The processing of this embodiment is presentationprocessing in the case where the magnification thresholds are set asshown in FIG. 16.

While the user is making an input operation for displaying a partialimage 210, the CPU 101 determines whether or not the output coordinategroup constituting the partial image 210 of the specified area includesat least one recorded coordinate of the plurality of recordedcoordinates in the table and its output magnification is equal to orlarger than its set magnification (Step 1003). In the case of Yes inStep 1003, the CPU 101 reproduces audio information associated with anID thereof (Step 1004).

With this, only after the output magnification of the partial image 210becomes equal to, for example, a magnification arbitrarily set by theuser, it is possible to reproduce the audio information.

Other Embodiment of Presentation Processing

In each of the above-mentioned embodiments of presentation processing,for example, in each of Steps 604, 804, 904, and 1004 in FIGS. 17, 22,26, and 27, the CPU 101 may execute, in addition to the step ofreproducing the audio information, the step of displaying the associatedmark image M.

Other Embodiment

The present application is not limited to the above-mentionedembodiments and other various embodiments may be realized.

In each of the above-mentioned embodiments, as the annotationinformation, mainly the audio information and the image such as the markimage M have been exemplified. However, in each of the above-mentionedembodiments, other types of annotation information described above maybe created and presented.

The PC 100 may include a program by which, without an input operation bythe user, a predetermined algorithm executes the specificationprocessing and the partial image 210 is automatically cut out and outputto the display unit 106. The PC 100 may include a program by which,without an input operation by the user, a predetermined algorithmautomatically generates mark information and the mark information isassociated with the output partial image 210.

In each of the above-mentioned embodiments, the user inputs theoperation information into the PC 100 via the controller 1 shown in FIG.4. However, other input devices such as a mouse and keyboard may be usedas a matter of course.

Although in each of the above-mentioned embodiments, the audioinformation is generated by the user as the mark information, the markinformation may be a symbol, a sign, an index, or other information thatare included in the PC in advance.

The present application is not limited to the applications in themedical and pathological fields. As long as it is a field in which thepartial image 210 is displayed at a plurality of differentmagnifications as an area specified within the huge entire image 200,the present application may be applicable to any fields. As such fields,the present application is applicable to not only biological, botanical,and bacteriological fields, but also geological, geophysical, and cosmicphysical fields.

Among the features of the above-mentioned embodiments, at least twofeatures may be combined together.

The present application may be configured as follows.

(1) A mark information recording apparatus, including:

an output unit configured to output, for displaying a partial imagebeing a part of an entire image having an arbitrary resolution amongentire images generated at a plurality of different resolutions withrespect to a single target, the partial image; and

a recording unit configured to associate mark information withinformation on a position within the entire image and information on theresolution of the output partial image and record the mark information.

(2) The mark information recording apparatus according to (1), in which

the recording unit is configured to record, as the mark information,annotation information and a mark image to be combined in the partialimage.

(3) The mark information recording apparatus according to (2), furtherincluding:

an annotation information generation unit configured to start, wheninformation on an input operation by a user for starting generation ofthe annotation information is acquired, the generation of the annotationinformation.

(4) The mark information recording apparatus according to (2), furtherincluding:

an annotation information generation unit configured to start, wheninformation on an audio of a speech of a user is acquired, generation ofthe annotation information.

(5) The mark information recording apparatus according to any one of (1)to (4), further including:

an acquisition unit configured to acquire information on an operation bya user for causing the output unit to output the partial image, in which

the recording unit is configured to associate the mark information withthe position information and the resolution information on the partialimage output by the output unit according to the operation informationacquired by the acquisition unit.

(6) The mark information recording apparatus according to (5), in which

the recording unit is configured to associate position information andresolution information with each of partial images different in at leastone of the position information and the resolution information, recordinformation obtained by the association as output history information oneach of the partial images, and associate the mark information with thehistory information, the partial images being output by the output unitaccording to information on a series of continuous or intermittentoperations by the user that is acquired by the acquisition unit.

(7) The mark information recording apparatus according to (5) or (6), inwhich

the recording unit is configured to execute, when, out of the operationinformation acquired by the acquisition unit, predetermined operationinformation is acquired, the association processing of the markinformation with respect to the partial image output according to thepredetermined operation information.

(8) The mark information recording apparatus according to any one of (1)to (7), in which

the recording unit is configured to associate priority informationindicating a priority upon presentation of each of a plurality of piecesof mark information with each of the plurality of pieces of markinformation and record the priority information.

(9) The mark information recording apparatus according to any one of (1)to (8), in which

the recording unit is configured to associate, with each of at least twoor more resolution ranges including the resolution of the partial imageoutput by the output unit, the mark information.

(10) The mark information recording apparatus according to any one of(1) to (9), in which

the recording unit is configured to set a threshold of the resolutionaccording to an operation by the user and to associate the set thresholdof the resolution with the position information and the mark informationand record the set threshold.

(11) A mark information presenting apparatus, including:

an output unit configured to output, for displaying a partial imagebeing a part of an entire image having an arbitrary resolution amongentire images generated at a plurality of different resolutions withrespect to a single target, the partial image;

a recording unit configured to associate mark information withinformation on a position within the entire image and information on theresolution of the output partial image and record the positioninformation and the resolution information that are associated with themark information as recorded position information and recordedresolution information, respectively; and

a processing unit configured to execute, when an output positioninformation group constituting the partial image to be output includesthe recorded position information, processing for presenting the markinformation associated with at least the recorded position information.

(12) The mark information presenting apparatus according to (11), inwhich

the processing unit is configured to execute, when the output positioninformation group includes the recorded position information and aresolution of output resolution information of the partial image to beoutput by the output unit is equal to a resolution of the recordedresolution information corresponding to the recorded positioninformation, processing for presenting the mark information associatedwith the recorded position information and the recorded resolutioninformation.

(13) The mark information presenting apparatus according to (12), inwhich

the recording unit is configured to record, as the mark information,annotation information and a mark image to be combined in the partialimage, and

the processing unit is configured to execute processing for presentingthe annotation information.

(14) The mark information presenting apparatus according to (13), inwhich

the processing unit is configured to combine, when the output positioninformation group constituting the partial image to be output includes aplurality of pieces of recorded position information, a plurality ofmark images associated with the plurality of pieces of recorded positioninformation, in the partial image to be output.

(15) The mark information presenting apparatus according to (14), inwhich

the processing unit is configured to combine, among a plurality ofpieces of recorded resolution information associated with the pluralityof pieces of recorded position information, one of an image having aresolution equal to the resolution of the output resolution informationof the partial image to be output by the output unit and an imagecorresponding to resolution information including a resolution within arecorded resolution range, in the partial image to be output.

(16) The mark information presenting apparatus according to (14), inwhich

the recording unit is configured to associate priority informationindicating a priority upon presentation of each of a plurality of piecesof annotation information corresponding to the plurality of images witha plurality of pieces of mark information and record the priorityinformation, and

the processing unit is configured to set, according to the recordedpriority information, one of whether or not to present the plurality ofpieces of annotation information and a time order of presenting theplurality of pieces of annotation information.

(17) The mark information presenting apparatus according to (14), inwhich

the recording unit is configured to associate, with each of at least twoor more resolution ranges including the resolution of the outputresolution of the partial image output by the output unit, theannotation information and set a different recorded resolution range foreach of different pieces of priority information, and

the processing unit is configured to execute processing for presentingat least one piece of annotation information among a plurality of piecesof annotation information according to the set recorded resolutionrange, the plurality of pieces of annotation information beingassociated with the plurality of pieces of recorded position informationincluded in the output position information group.

(18) The mark information presenting apparatus according to any one of(11) to (17), further including:

an acquisition unit configured to acquire information on an operation bya user for causing the output unit to output the partial image, in which

the recording unit is configured to execute, when, out of the operationinformation acquired by the acquisition unit, predetermined operationinformation is acquired, the association processing of the markinformation with respect to the partial image output according to thepredetermined operation information, and

the processing unit is configured to execute, when an output positioninformation group constituting the partial image output according to thepredetermined operation information includes the recorded positioninformation, processing for presenting the mark information associatedwith at least the recorded position information.

(19) A mark information recording method, including:

outputting, for displaying a partial image being a part of an entireimage having an arbitrary resolution among entire images generated at aplurality of different resolutions with respect to a single target, thepartial image; and

associating mark information with information on a position within theentire image and information on the resolution of the output partialimage and recording the mark information.

(20) A mark information presenting method, including:

outputting, for displaying a partial image being a part of an entireimage having an arbitrary resolution among entire images generated at aplurality of different resolutions with respect to a single target, thepartial image;

associating mark information with information on a position within theentire image and information on the resolution of the output partialimage and recording the position information and the resolutioninformation that are associated with the mark information as recordedposition information and recorded resolution information, respectively;and

executing, when an output position information group constituting thepartial image to be output includes the recorded position information,processing for presenting the mark information associated with at leastthe recorded position information.

The above-mentioned mark information presenting apparatus may beconfigured as follows.

(21) The mark information presenting apparatus according to (18), inwhich

the recording unit is configured to associate position information andresolution information with each of partial images different in at leastone of the position information and the resolution information, recordinformation obtained by the association as history information includingthe recorded position information and the recorded resolutioninformation of each of the partial images, and associate the markinformation with the history information, the partial images beingoutput by the output unit according to information on a series ofcontinuous or intermittent operations by the user that is acquired bythe acquisition unit, and

the processing unit is configured to execute, when the output positioninformation group constituting each of the partial images to be outputby the output unit includes at least one piece of recorded positioninformation included in the history information, processing forpresenting the mark information associated with the history information.

(22) The mark information presenting apparatus according to any one of(11) to (18) and (21), further including:

an acquisition unit configured to acquire information on an operation bythe user for causing the output unit to output the partial image, inwhich

the recording unit is configured to associate the mark information withthe position information and the resolution information on the partialimage output by the output unit according to the operation informationacquired by the acquisition unit and record the position information andthe resolution information that are associated with the mark informationas the recorded position information and the recorded resolutioninformation, respectively; and

the processing unit is configured to execute, when the output positioninformation group of the partial image to be output by the output unitaccording to the operation information acquired by the acquisition unitincludes the recorded position information and the resolution of theoutput resolution information of the partial image to be output by theoutput unit is equal to the resolution of the recorded resolutioninformation corresponding to the recorded position information,processing for presenting the mark information associated with therecorded position information and the recorded resolution information.

(23) The mark information presenting apparatus according to any one of(11) to (18), (21), and (22), in which

the recording unit is configured to associate, with each of at least twoor more resolution ranges including the resolution of the outputresolution information of the partial image output by the output unit,the mark information and record information on the resolution range withwhich the mark information is associated, as information on recordedresolution range, and

the processing unit is configured to execute, when the resolution of theoutput resolution information falls within the recorded resolutionrange, processing for presenting the mark information corresponding tothe output partial image.

(24) The mark information presenting apparatus according to (11), inwhich

the processing unit is configured to execute, when the output positioninformation group includes a plurality of pieces of recorded positioninformation, processing for presenting a plurality of pieces of markinformation associated with the plurality of pieces of recorded positioninformation.

(25) The mark information presenting apparatus according to (13), inwhich

the processing unit is configured to execute, according to informationon an operation by a user using the mark image, processing forpresenting the annotation information corresponding to the mark image.

(26) The mark information presenting apparatus according to any one of(11) to (18) and (21) to (23), in which

the recording unit is configured to set a threshold of the resolutionaccording to an operation by the user and to associate the set thresholdof the resolution with the position information and the mark informationand record the set threshold, and

the processing unit is configured to execute, when the resolution of thepartial image to be output by the output unit exceeds the set threshold,processing for presenting the mark information corresponding to thepartial image to be output.

It should be understood that various changes and modifications to thepresently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications can be madewithout departing from the spirit and scope of the present subjectmatter and without diminishing its intended advantages. It is thereforeintended that such changes and modifications be covered by the appendedclaims.

1-8. (canceled)
 9. A signal processing apparatus comprising: circuitryconfigured to obtain a position information of an image and a firstmagnification level of the image, and output an annotation information,wherein the annotation information is a first annotation informationcorresponding to the position information and a second magnificationlevel equal to or larger than the first magnification level.
 10. Theapparatus according to claim 9, wherein the image is an image of anorgan, a tissue, or a cell of a living body.
 11. The apparatus accordingto claim 9, wherein the annotation information is a text data, an audiodata, an image data, or a link data.
 12. The apparatus according toclaim 11, wherein the audio data includes speech of a user.
 13. Theapparatus according to claim 9, wherein a form of the annotationinformation depends on a user.
 14. The apparatus according to claim 9,wherein a form of the annotation information depends on an informationtype.
 15. The apparatus according to claim 14, wherein the informationtype is one of a text data, an audio data, an image data, or a linkdata.
 16. The apparatus according to claim 9, wherein the annotationinformation is output based on a priority information.
 17. The apparatusaccording to claim 16, wherein the priority information is one of animportance of the annotation information, a user information whoattached the annotation information, or a time when the annotationinformation attached.
 18. A signal processing method comprising:obtaining a position information of an image and a first magnificationlevel of the image, and causing a display device to display anannotation information, wherein the annotation information is a firstannotation information corresponding to the position information and asecond magnification level equal to or larger than the firstmagnification level.
 19. A method comprising: obtaining a request of afirst area of an image; and causing a display device to display thefirst area of the image and a first annotation information, wherein thefirst annotation information is the annotation information correspondingto coordinate groups of the first area and a magnification equal to orlarger than the magnification of the first area.